The Internet has emerged as a critical communication infrastructure, carrying traffic for a wide range of important scientific, business and consumer applications. Network service providers and enterprise network operators need the ability to detect anomalous events in the network, for network management and monitoring, reliability, security and performance reasons. While some traffic anomalies are relatively benign and tolerable, others can be symptomatic of potentially serious problems such as performance bottlenecks due to flash crowds, network element failures, malicious activities such as denial of service attacks (DoS), and worm propagation. It is therefore very important to be able to detect traffic anomalies accurately and in near real-time, to enable timely initiation of appropriate mitigation steps.
One of the main challenges of detecting anomalies is the mere volume of traffic and measured statistics. This is a particular challenge where the system architecture does not leverage such methods as built-in bottlenecks for failsafe enforcement of policy controls. Given today's traffic volume and link speeds, the input data stream can easily contain millions or more of concurrent flows, so it is often impossible or too expensive to maintain per-flow state. The diversity of network types further compounds the problem. Thus, it is infeasible to keep track of all the traffic components and inspect each packet individually for anomalous behavior. Further risks include the difficulty in discerning whether a usage pattern constitutes the unauthorized access, control or modification of information or system resources. Host-based and network-based logging provides a potential recognition basis as well as the forensic capability to ensure a level of accountability for action or inaction.
Another challenge is that different types of anomalies manifest themselves in a variety of ways and remain in the network for different durations. The anomalies with large durations are identified by detection methods such as top ten counting. The anomalies that are a major challenge to detect are those appearing repeatedly for short durations. Another challenge is the unauthorized tunneling or copying of information, or example by malfeasant information gathering, illicit proxy or store/forward, hijacked management capabilities or outright spyware.
Therefore, a need exists for a method and apparatus for near real-time detection of anomalies in traffic logs that elude simple ranking methods such as “top ten” counting. Anomaly detection is critical for monitoring and maintaining packet networks, e.g., Voice over Internet Protocol (VoIP) networks.